Software performance testing detects software code changes that may have slowed down the execution of an application, which may be referred to as performance regressions. The application slowdowns may have a negative effect on user experience while using the application, and can therefore deter many users from continuing to use the application.
Finding the cause of the performance regression, referred to as a performance regression cause, is a very time-consuming process when performed manually. For example, typically, execution timelines for the two versions of the application need to be generated by observing one version before the performance regression and observing another version after the performance regression. An execution timeline may be a call tree describing the functions that were executed while the user was interacting with the application where the call tree is augmented with information on how long each function took to execute. Then, a user must analyze and compare the performance results included in the execution timelines. For example, the user would usually have to compare the functions in the call trees one-by-one, recursively going down the call trees until the lowest level function that regressed is discovered. Knowing the lowest level function may provide a strong lead to the code change that caused the regression or contains the code change itself.
The above process is further complicated by variance in the performance numbers, which often necessitates multiple execution timelines to be generated per version and hence for multiple timelines to be analyzed and compared. It is further difficult to conduct the above analysis when performance regressions are observed at the client side of web applications due to challenges associated with software code being executed, such as JavaScript™. For example, web applications rely on features that exploit the asynchronous nature of the software code (e.g., timers, promises, callbacks, etc.), which can lead to large execution timelines involving multiple call trees—one for each asynchronous execution. These timelines are very difficult for the user to compare manually given their size. Further, JavaScript™ is an event-driven language, which means simple user interactions (e.g., mouse outs, hovers, scrolls, etc.) may trigger event handlers that may be irrelevant to the performance regression, thereby introducing noise to the execution timelines being compared. Accordingly, performing the analysis manually may be time-consuming and also prone to errors due to the complexity of the analysis. Realistically, the analysis could not be performed manually in an acceptable time limit or at all.